Created overview table for tasks and where they're used. Also version for google sheets https://docs.google.com/spreadsheets/d/1wyTvW0q6TIat7RMmfimlNKXri9O7cs_S0uebGTNya0c/edit?usp=sharing

import mteb
import pandas as pd

tasks = mteb.get_tasks(modalities=["audio"])

audio_tasks_names = [t.metadata.name for t in mteb.get_benchmark("MAEB(audio)")]
audio_text_tasks_names = [t.metadata.name for t in mteb.get_benchmark("MAEB(audio-text)")]

row = []
for task in tasks:
    print(task.metadata.name)
    in_audio = task.metadata.name in audio_tasks_names
    in_audio_text = task.metadata.name in audio_text_tasks_names
    row.append(
        {
            "Task Name": task.metadata.name,
            "Task description": task.metadata.description,
            "Task type": task.metadata.type,
            "Task language(s)": ", ".join(task.metadata.eval_langs) if isinstance(task.metadata.eval_langs, list) else ", ".join(v[0] for v in task.metadata.eval_langs.values()),
            "In MAEB(audio)": "Yes" if in_audio else "No",
            "In MAEB(audio-text)": "Yes" if in_audio_text else "No",
        }
    )

df = pd.DataFrame(row)
df = df.sort_values(by=["Task Name", "Task type"]).reset_index(drop=True)
df.to_csv("audio_tasks_table.csv", index=False)
df.to_markdown("audio_tasks_table.md")

Probably we can create english only version, but I'm not sure if it is relevant, because most of the tasks are english only

Where are all the multilingual tasks?

I think we can create

1. MAEB(audio)
2. MAEB(audio-text-multilingual)
3. MAEB(audio-text-eng)

But this might be complicated to understand for users

I think we can create

1. MAEB(audio)
2. MAEB(audio-text-multilingual)
3. MAEB(audio-text-eng)

But this might be complicated to understand for users

Why would it be complicated? Seems clear to me

Hmm I would maybe do:

• MAEB: the full MAEB, including audio, audio-text and multilingual
• MAEB(audio): The audio-only subset of MAEB
• MAEB(english): The english subset of MAEB

However, I would probably argue we could just make two columns that are audio-only and English and just maintain a single benchmark. WDYT? This both simplifies use, the selection and the paper itself.

PS: We have to fix the language annotations - birdset for example, is not English.

We have to fix the language annotations - birdset for example, is not English

How we should name it? Just other or you have something specific in mind? We probably need to change it also to GTZAN (music classification), GunshotTriangulation, MridinghamTonic, NSynth tasks. Added an issue #3872

However, I would probably argue we could just make two columns that are audio-only and English and just maintain a single benchmark. WDYT? This both simplifies use, the selection and the paper itself.

For leaderboard, I agree, but for the users I'm not sure because this can create problems on inference

we could just make two columns that are audio-only and English and just maintain a single benchmark.

Sorry it's been a long day, and for some reason I struggle to envision this. What would this look like? Would this need any change to the LB?

Ah I get it now, only maintain MAEB. Do we bother filtering out similar tasks? or use the entire collection?

I'm a bit afraid that if we use only 1 benchmark, but users would want to evaluate only on part of it, e.g. audio only. They would need to filter tasks

What if we have an english list, an audio list, a "the rest of the collection" list, and MAEB is english + audio + "the rest"? We can still have MAEB(eng)v1, MAEB(audio)v1, and MAEBv1 ?

I resolved #3877 and removed zeroshot tasks from the audio-only benchmark.

Just added the MAEB audio extended and lite benchmarks. These have 54 tasks, 38 models and 19 tasks, 44 models respectively. MAEB audio-text lite has 30 tasks, 10 models.

This is done by finding the most number of tasks with the most number of model eval runs completed. No filtering is applied. @AdnanElAssadi56 @KennethEnevoldsen @Samoed would love a quick pair of eyes on these. I'd say we can probably start with these, and start filling in the relevant paper subsections.

Audio, Extended

Audio, Lite

Audio-Text, Lite

Look great!

Great work, @isaac-chung!

When we say audio-text, lite here, are we implying an extended version to the readers?

Great work, @isaac-chung!

When we say audio-text, lite here, are we implying an extended version to the readers?

It's the most complete collection bases on what we have run. We're missing results for an extended version.

Added MAEB(audio-text, extended) as well. Removed same task A2T variants from MAEB(audio-text, lite) based on our prev discussion:

Here we can prioritize the direction that seems the most valid (e.g. T2A might be more valid than A2T in some cases), or based on unique attributes (task A is the only one that covers domain X so we keep that one) or runtime (task A is much faster to run that task B so we choose A).

@AdnanElAssadi56 @Samoed @KennethEnevoldsen I've update both this branch AND the paper draft based on the following:

MAEB Benchmark Summary

Notes:

• Lite benchmarks achieve 2–9× speedup vs extended variants
• Lite-to-extended correlation: Spearman ρ=0.90 (audio-only), ρ=1.00 (audio-text)
• Task types for audio: Classification, Clustering, PairClassification, Reranking, Retrieval
• Task types for audio-text: Retrieval (A2T/T2A), ZeroshotClassification

Great work! Maybe we can create versions with only English?

Thanks!

I feel our recurring theme overall has been maintainability - and that drives us to keep the number of benchmarks low. As such, I feel a modality split is the only key factor to have separate benchmarks. This way, we can also make a claim that since it's inherently multilingual, we incentivize/nudge the community to develop better multilingual audio embedding models.

For English subsets, perhaps we only show a doc example on how to filter tasks?

I'm a bit afraid that if we use only 1 benchmark, but users would want to evaluate only on part of it, e.g. audio only. They would need to filter tasks

I was thinking this would be the ideal behaviour. We could easily add code on that (we could even add the benchmarks), but without adding multiple views in the leaderboard.

I really agree with:

I feel our recurring theme overall has been maintainability

Which is why I would rather have a single benchmark with filters. I think this aligns fairly well with what we have now though. This is how I would phrase it in the paper. We construct a broad range of tasks. We call this collection MAEB+. This is the unreduced set - extended. Then we actual benchmark is a condensed version of this (MAEB+ never becomes a released benchmark it is just a collection of tasks used to construct MAEB). What do you guys think?

I am unsure if we want to keep audio and audio-text seperated though. Here I am leaning combine, but it is only a small preference though (will look more at the paper to figure out what is best)

inherently multilingual, we incentivize/nudge the community to develop better multilingual audio embedding models.

I agree that default multilingual (and potentially default multimodal, audio-text?) is a good incentive to provide. People will be interested in the English column, but we can provide that.

Questions

• What does ~3 languages (multilingual) mean?
• Seems like we loose a lot of languages in the lite in audio-text. Seems concerning to me.
• We also loose a task type in the lite version of audio, which I also think is problematic (we could do a similar check on domains)

(will look more in the paper as well)

Overall, I think we can include English only (or multilingual) benchmark without zxx languages, because some models could have low performance on zxx tasks, but higher performance on real real languages.

I feel our recurring theme overall has been maintainability

What is the problem with maintainability?

Modality split seems the most practical still: There are just a lot more audio-only embedding models, and fewer audio-text-capable models.

I agree that default multilingual (and potentially default multimodal, audio-text?) is a good incentive to provide. People will be interested in the English column, but we can provide that.

💯 I think an English column and a zxx column would complete the picture really well.

What is the problem with maintainability?

High number of benchmarks lowers maintainability.